Papers by Erland Hilman Fuadi
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax (2026.findings-acl)
Copied to clipboard
| Challenge: | Quantized models using softpick outperform softmax on standard benchmarks . softmax is widely used in statistics and especially in machine learning . |
| Approach: | They introduce a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. |
| Outcome: | The proposed model outperforms softmax on benchmarks with lower bit precisions. |